{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cd88c9a4",
   "metadata": {},
   "source": [
    "# Task Completion Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates how to use task completion evaluator on agent data. The supported input formats include:\n",
    "- simple data such as strings and `dict` describing task responses;\n",
    "- user-agent conversations in the form of list of agent messages. \n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe5dda1c",
   "metadata": {},
   "source": [
    "The Task Completion evaluator assesses whether an AI agent successfully completes the requested task by examining:\n",
    "- Whether the task was fully completed\n",
    "- Quality of task execution\n",
    "- Appropriateness of the response to the original request\n",
    "\n",
    "The evaluator uses a binary scoring system (0 or 1):\n",
    "\n",
    "    - Score 0: The task was not completed or only partially completed\n",
    "    - Score 1: The task was successfully and fully completed\n",
    "\n",
    "This evaluation focuses on measuring whether the agent's response indicates successful completion of the user's request, regardless of the specific methods or tools used to achieve the task."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1642178",
   "metadata": {},
   "source": [
    "Task Completion requires following input:\n",
    "- Query - This can be a single query or a list of messages(conversation history with agent). The original task request from the user.\n",
    "- Response - Response from Agent (or any GenAI App). This can be a single text response or a list of messages generated as part of Agent Response.\n",
    "- Tool Definitions - (Optional) Tool(s) definition used by Agent to answer the query. Providing tool definitions helps the evaluator better understand the context and capabilities available to the agent.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffad775f",
   "metadata": {},
   "source": [
    "### Initialize Task Completion Evaluator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a5fae33",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation._evaluators._task_completion import _TaskCompletionEvaluator\n",
    "from azure.ai.evaluation import AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "\n",
    "task_completion = _TaskCompletionEvaluator(model_config=model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af4839bb",
   "metadata": {},
   "source": [
    "### Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95305bbd",
   "metadata": {},
   "source": [
    "#### Evaluating Simple Task Completion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57d4592d",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "response = \"The current weather in Seattle is partly cloudy with a temperature of 15°C (59°F). There's a light breeze from the northwest at 8 mph, and the humidity is at 68%. No precipitation is expected for the rest of the day.\"\n",
    "\n",
    "# Basic evaluation without tool definitions\n",
    "result = task_completion(query=query, response=response)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c66a362",
   "metadata": {},
   "source": [
    "#### Task Completion with Tool Context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71ea8725",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "response = \"I've checked the weather for Seattle and found that it's currently partly cloudy with a temperature of 15°C. There's a light breeze and no rain expected today.\"\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "result = task_completion(query=query, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1962234d",
   "metadata": {},
   "source": [
    "#### Task Completion with Tool Definition as Single Dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a9cd459",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What's the current temperature in Boston?\"\n",
    "response = \"The current temperature in Boston is 22°C (72°F) with clear skies. It's a beautiful day with low humidity and a gentle breeze.\"\n",
    "\n",
    "tool_definition_dict = {\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "    },\n",
    "}\n",
    "\n",
    "result = task_completion(query=query, response=response, tool_definitions=tool_definition_dict)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc85ade7",
   "metadata": {},
   "source": [
    "#### Complex Task with Multiple Steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "915768db",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Can you send me an email with weather information for Seattle?\"\n",
    "response = [\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"I'll get the current weather information for Seattle and then send you an email with the details.\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:42Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle. The current weather is partly cloudy with a temperature of 15°C. You should receive the email shortly at your registered email address.\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = task_completion(query=query, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5523dfcf",
   "metadata": {},
   "source": [
    "#### Query as Conversation History (List of Messages)\n",
    "The evaluator also supports query as a list of messages representing conversation history. This helps evaluate task completion in the context of a full conversation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c4772be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Query as conversation history instead of a single string\n",
    "query_as_conversation = [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\", \n",
    "        \"content\": \"Hi, I need to plan my day. Can you check the weather in Seattle for me?\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": \"Also, please send me an email summary of the weather so I can reference it later.\"\n",
    "    }\n",
    "]\n",
    "\n",
    "response = \"I've checked the weather in Seattle for you. It's currently 15°C and partly cloudy with light winds. I've also sent you an email summary with these details so you can reference them throughout your day. The task has been completed successfully.\"\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = task_completion(query=query_as_conversation, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "100e943b",
   "metadata": {},
   "source": [
    "#### Example of Incomplete Task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9986fa70",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Can you send me an email with weather information for Seattle?\"\n",
    "incomplete_response = \"I can see that you want weather information for Seattle. The weather there is usually quite nice this time of year, with temperatures ranging from mild to warm depending on the season.\"\n",
    "\n",
    "# This response doesn't complete the email sending task\n",
    "result = task_completion(query=query, response=incomplete_response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
